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ABSTRACT 

The concept of Open Education (OE) is based on the philosophy of e-Learning which aims to provide learning 
environment anywhere, anytime, and for anyone. One of the main issue in the development of OE services is the 
availability of the quality assurance mechanism. This study proposes a metric for measuring the quality of OE 
service. Based on extensive literature review about e-Learning service quality guidelines, as well as the 
observation of existing Open Education services, we formulated Open Education Metric (OEM). OEM consists 
of 37 indicators which are divided into six criteria: (1) Openness, (2) Benefit, (3) Delivery, (4) Learning, (5) 
Evaluation, and (6) Support. Evaluation scheme is designed to give a clear guideline in assessing the quality of 
OE service quantitatively. The evaluation results including score and category indicate the overall quality 
relative to a set of indicators. Trial of evaluation was conducted using the evaluation system based on OEM to 
test its performance in evaluating OE service quality in institutions. Three of OE services were evaluated with 
their scores. 

Keywords: metrics; online learning; open education; service quality 

INTRODUCTION 

Open Education was first introduced by David Wiley in 2011 (Pisutova, 2012). Open Education has the 
characteristics of e-Learning where learning activities can be done at anytime and anywhere, coupled with the 
concept for anyone. The application of the concept of Open Education emphasizes on the openness of access to 
education for everyone in a large scale, for example, is a service Massive Open Online Course (MOOC). The 
application of the concept of Open Education varies greatly. It is not just limited to the MOOC. According 
Pisutova (2012) there are at least four categories of services: Open Content, Open Courseware, Open 
Educational Resources, and Open Teaching. Categorization is done by what is offered by a service. Variations 
which occur in more detail services, such as certification, the structure of lectures, presentation of lectures, open 
access, standards related to the lecture material, availability of information related to the course, the availability 
of technical assistance, availability of tutors in the learning and so forth. 

In the context of Open Education, the absence of a model as a reference will make it hard for an organization to 
plan the steps to be taken to implement the service (Yuan, MacNeill, & Kraan, 2008). It also requires an 
evaluation mechanism to measure the extent to which the implementation of Open Education has been done, so 
it can be compared to other services. Prospective users of the service will be better informed if there is a large 
selection of quality benchmark information service. 

Some maturity models related to e-learning and software engineering in general have ever been studied 
previously, for example, the Capability Maturity Model (Paulk, Curtis, Chrissis, & Weber, 1993), e-Learning 
Maturity Model (Marshall & Mitchell, 2002), e-Learning Process Maturity Model (Zhou, 2012), and Online 
Course Quality Maturity Model (Gu, Chen, & Pu, 2011). All of the maturity models are designed as the standard 
in the implementation process with the main purpose of guaranteeing and promoting the quality of the products 
or services produced. Maturity models provide guidance and the ideal standard of the results expected from the 
implementation of the process. If maturity models provide a thorough benchmark based on best and ideal 
practice, there is also a metric that does not directly provide ideal standards regarding expected results of a 
process. Metric emphasizes more on the evaluation of a process of the indicators that are designed and can 
measure the quality of the results. Some of the metrics that have already existed and are related to this research 
among which the e-Government Metric and the Web-based Application Quality Metric. 

However, if the reference puts much emphasis on the aspect of a well-executed and documented policies or 
procedures that govern the process with a high level of complexity which is commonly found on maturity 
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models, it can potentially discourage the institutions initiative to implement the Open Education service. In 
addition, the measurement or benchmarking mechanism involving internal factors/social service providers is not 
always easy to do, for example, the institutional readiness factor in implementing e-Learning, the regularity of 
the process of designing the curriculum, user satisfaction, and so on. in other words, we need a reference 
(simple) which can also be used as a measuring instrument (metric) practical application of Open Education 
(practical) based on the facts that can be accessed via the internet. 

Therefore, taking into account the need for quality assurance of Open Education services (Hylen, 2006; Yuan, 
MacNeill, & Kraan, 2008; Yuan & Powell, 2013), this study focuses to answer the following three research 
questions: 

1. What are the indicators used in assessing the quality of service of Open Education? 

2. How to make an evaluation based on these indicators? 

Based on the research problem, a few objectives have been defined to be achieved from this research. The first is 
to identify indicators in assessing the quality of service of Open Education. The second is to formulate the Open 
Education metric for evaluating Open Education service. Open Education Metric (OEMs) are expected to 
contribute as a reference for the development and evaluation of services Open Education. 

There are many aspects that can be extracted from the application of Open Education, and certainly not all 
aspects are discussed in this study. Based on the research objectives that have been set, taking into account the 
aspect of simplicity and practicality, we would then determine the limits of the research coverage. Here are some 
boundaries that define the scope of this study: 

1. Criteria which will be discussed in the research are those covering the external/technical aspects 
(instructional presentation, open access, evaluation, technical assistance etc.), and not the internal/social 
aspects (institutional readiness, a standard procedure of education material production, institutional 
satisfaction, user satisfaction, and so on.) on the application of Open Education. 

2. The indicators that have been selected can be observed directly through the internet for the services 
concerned. This study refers to the method used by Waseda e-Government Ranking in which the 
assessment is based upon the characteristics of e-Government services that can be accessed via the 
Internet independently. 

RELEVANT LITERATURE REVIEW 
E-Learning and Open Education 

One of the definitions of e-Lea rning from previous researchers is the combination of educational functions and 
the provision of teaching materials through information technology or the Internet (Tzeng, Chiang, & Li, 2007). 
E-Leaming is widely used by educational institutions as a reference to implement distance education, which 
make it synonymous with the term distance learning (Welsh et ah, 2003). An educational service that 
implements e-Learning has a set of components that are closely interwoven. Based on research Selviandro 
(2013) regarding the service cloud-based e-Learning, it can be concluded that the main components that make up 
an educational service based e-Learning are an actor, data, applications, and infrastructure. 

The concept of e-Learning that use information technology in the process of teaching and learning has changed 
the face of education, especially for such high levels of education at the university. Starting from the concept of 
e-Learning, we now apply the paradigm of Open Education. The basic philosophy underlying the Open 
Education is the same as the underlying e-learning, which is learning anytime and anywhere, but Open 
Education also greatly emphasizes the aspects 'for anyone'. It represents the philosophy of openness and sharing 
which are also an integral part of the process of formation and dissemination of knowledge (Wiley, 2011). 
Therefore, Open Education aims to expand access to knowledge to all corners of the world without being limited 
to differences in geographical, economic and social aspects, with Internet technology as the primary means. 

Figure 1 illustrates the integrated concepts that are part of the paradigm of Open Education, the Open Content, 
Open Courseware, Open Educational Resources and Open Teaching (Pisutova, 2012). Arrows indicate the 
direction of development of the concept, which initially only emphasizes the sharing of learning materials 
(content sharing) on the concept of Open Content, which then becomes an online educational service that is 
integrated in the concept of Open Teaching. The development of this concept towards the implementation of 
Open Education which has a wider reach and offer more benefits to its users. 
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Figure 1. Paradigm of Open Education (adapted from Pisutova in 2012) 


Furthermore, based on the existing definitions (Pisutova, 2012; Wiley, 2011), some aspects can be inferred to 
distinguish between the concepts of Open Content, Open Courseware, Open Educational Resources and Open 
Teaching. Table 1 shows two aspects that distinguish between these concepts, namely from the aspect of services 
presented in each concept and the benefits to be gained by an individual of service on each concept. Each 
concept has its own character in every aspect, and an enrichment and the refinement of the concept of the 
previous level. 


Table 1. The difference between the concepts of the Open Education 


Aspect 

Open Content 

Open Courseware 

Open Educational 
Resources 

Open Teaching 

Service 

Learning materials, 
for example in the 
form of text, video 
and animations 

Learning material 
presented in the 
currency of 
teaching 

Courseware coupled 
with other learning 
resources, such as tools 
and infrastructure 

Online educational 
services, including 
materials and 
evaluation of learning 

Benefits for 
Individuals 

Science 

Set science on a 

course 

Science enriched by 
learning from the 
experience of other 
learning resources 

Certification or 
college credit 


E-Learning Maturity Model 

Based on the work of a pioneer in the field of software engineering named Watts Humphrey, Paulk et al. (1993) 
creates a Capability Maturity Model (CMM). CMM is the solution to improve the quality and productivity of 
software development. CMM provides instructions for resolving the challenges caused by the inability of an 
organization to define and implement a systematic and structured development process. Based on benefits that 
can be obtained by applying the Capability Maturity Model, Marshall and Mitchell (2002) conducted a study to 
devise a maturity model for improving the quality of e-Learning. Thus, in principle, the E-Learning Maturity 
Model (EMM) is a modification of the CMM specifically to the field of information technology-based education 
items, namely e-Learning. Table 2 describes the objectives that are the focus at any level or degree of maturity 
EMM. 


Table 2. Focus of Each Level in e-Learning Maturity Model 


Level 

Focus 

0. Not Performed 

There is no implementation of e-Learning 

1. Initial 

The process is still random, improvised or ad-hoc 

2. Repeatable 

E-Learning Objectives are clearly defined 

3. Defined 

Process development and operation of e-Learning clearly defined 

4. Managed 

Quality control over materials and learning outcomes of e-Learning 

5. Optimizing 

Continuous improvement effort on the quality of e-Learning 


METHODOLOGY 

The development stages in this study are generally divided into four phases: the requirement identification, 
analysis, design, and evaluation. This partition is intended to provide a roadmap so that research can be done in a 
structured and systematic way. A number of activities are conducted at each phase sequentially to avoid overlap, 
following the workflow of logical thinking by deductive reasoning. The workflow of the stages of this study is 
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illustrated in Figure 2. 


Identification of Metric Requirements 


Analysis of Criteria, Indicator, and Evaluation Scheme 
in Assessing Quality of Open Education Service 


Design of Open Education Metric 


Testing and Evaluation 


Figure 2. Research Workflow 


Identification of Metric Requirements 

At this stage we conduct a literature study, the observation and analysis aim to identify the things that are 
necessary out of a metric. This stage focuses on the existing standard of quality measurements (metric) which is 
related to the field of Open Education. Literature review is applied to better understand the number of concepts 
concerning the application of Open Education in the institutions. It also includes the observation of'best practice' 
in Open Education services that already exist, for example the MIT Open Courseware, Coursera, EDX, Udacity, 
Futurelearn, and so forth. The analysis, synthesis, and interpretation have been made based on the literature and 
best practice. The results of the analysis are in the form of a number of characteristics that should be possessed 
by a metric created. 

Analysis of Criteria, Indicators and Evaluation Scheme in Assessing Quality of Open Education Service 
This stage is to determine the criteria that can represent the quality of Open Education service. Selection of 
criteria is one of the most crucial step in this study because it provides a foundation for the design metrics. The 
method used at this stage includes the analysis, synthesis, interpretation of literature, and observation of Open 
Education services that already exist as the best practice. The literature considered includes the publications in 
the area of Open Education, e-Learning, maturity models, metrics and evaluation of e-Learning. The literature 
review has been conducted to provide a reference and basic theory in the preparation of metrics. 

Design of the Open Education Metric 

Furthermore, based on the criteria and potential indicators that have been acquired, we can design an evaluation 
model or scheme of quantitative assessment to measure the quality of service of Open Education. Broadly 
speaking, the resulting metric is the result of the analysis, synthesis and interpretation of the results of the 
analysis in the previous stage. The results of the Open Education Metric design consist of criteria, indicators and 
evaluation schemes for measuring service quality of Open Education. 

Testing and Evaluation 

The trial of metrics is done assessing the quality of some Open Education services in the real world. The results 
of these tests are used to analyze Metric Open Education qualitatively. Furthermore, to provide an overview 
about the relevance and validity of indicators used in assessing OE service quality, expert and user judgment (5- 
10 people) was conducted. This also aims to reduce subjectivity and bias in the development of metrics, as well 
as giving additional perspective from the expert and user viewpoint. In this study, an expert is defined as those 
who have experience in e-Learning as a researcher, while also being familiar with the existing growth and 
features of Open Education services. In contrast, the user is defined as those who have experience in using e- 
Leaming and Open Education services, but is not familiar with the e-Learning research field. Academic 
qualifications required for all experts and users is a Master degree holder, in order to ensure they have an 
adequate cognitive level that is required in analyzing the factors that affect the OE service quality. 

FINDINGS 
Metric Requirements 

The concept of openness to educational resources is not new, and a lot of institutions have participated in the 
sharing of educational resources, such as MIT Open Courseware since the early 2000s. The growing interest in 
the Open Education concept, as well as the increasing awareness of its benefit to all parties involved, leads to 
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development of Open Education services in number. However, with the number of services that have been or 
will appear in the future, we need a guideline to ensure the quality of this services (Hylen, 2006; Yuan, 
MacNeill, & Kraan, 2008; Yuan & Powell, 2013). 

Maturity model is an example of a guideline that is generally interpreted qualitatively, which contains standard 
guidelines in the implementation process, the ideal characteristics, and the ideal results expected from a process. 
Meanwhile, one example of the guidelines that are usually analyzed quantitatively is metric, with a focus on 
measuring the quality of observed characteristics quantitatively. The research was conducted by considering a 
number of maturity models and metrics that have already existed, namely in the field of software development, 
e-Leaming, and e-Government. 

Table 3 summarizes a number of problems and needs to be addressed by a metric, in order to assess the quality 
of Open Education service. Therefore, Open Education Metric will be formulated based on this set of problems 
and needs. 


Table 3. Requirements Identification of the Open Education Metric 


Problem 

Need 

Solution 

Institutions do not understand the concept of service 
in detail but would like to try to implement the 
service as a pilot project 

Practical guidelines for the 
implementation of Open Education 
services 

Open 

Education 

Metric 

Lack of standard implementation guideline for 
service provider or prospective service providers 

Examples of standard implementation 

Service provider’s difficulties in evaluating the 
implementation that has been done 

Evaluation standard for service quality 
across relevant aspects 

The absence of a mechanism to compare the quality 
of a service with other services (benchmarking) 

The mechanism to measure the quality 
of relevant aspects quantitatively 

Lack of specific guideline regarding the quality of 
Open Education service; although there are many 
references for the quality of e-Learning, we need to 
consider if all is relevant and to be implemented 

Guidelines about the quality of the 

Open Education service by utilizing the 
e-Learning quality benchmark that is 
readily available and relatively well- 
established, as its base 

Users and providers who usually prefer the practical 
aspect is difficult to interpret qualitative evaluations 

The mechanism to quantitatively 
evaluate the service quality 

Guideline about the quality of e-Learning that is 
intuitively also applicable for Open Education, still 
too abstract and too broad for interpretation 

Open Education service quality 
guidelines with practical indicators and 
clear directions 


Key Criteria and Indicator of Open Education Service Quality 

There are many different criteria that can represent several aspects of Open Education services, which can be 
measured and considered as the quality of a particular service from the user perspective. Among numerous 
criteria found on literatures (e.g., Blumberg, 2009; CSU, 2009; Gu, Chen, & Pu., 2011; IHEP, 2000; Khan, 2001; 
Marshall & Mitchell, 2002, 2004; Ming-Li & Dan, 2011; Moore, 2005; Pisutova, 2012; Price, Richardson, & 
Jelfs, 2007; Shelton, 2011; Wiley, 2011), there are six main criteria that we proposed and considered to be the 
focus of this study: Openness, Benefit, Delivery, Learning, Evaluation, and Support. Each criterion represents 
the quality of service of Open Education from different aspects. Table 4 illustrates reference from which each 
criterion synthesized. 
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Table 4. Literature Reference of Key Criteria 


Literature Reference 

Openness 

Benefit 

Delivery 

Learning 

Evaluation 

Support 

(Blumberg, 2009) 







(CSU, 2009) 







(Wiley, 2011) 







(Gu, Chen, & Pu., 2011) 







(IHEP, 2000) 







(Khan, 2001) 







(Marshall & Mitchell, 2002) 







(Marshall & Mitchell, 2004) 







(Ming-Li & Dan, 2011) 







(Moore, 2005) 







(Pisutova, 2012) 







(Price, Richardson, & Jelfs, 2007) 







(Shelton, 2011) 








Then, we conduct an analysis of indicators related to each criterion. Each indicator synthesized from extensive 
literature review of previous work related to e-Learning, while also considering the nature of Open Education 
concept. Characteristics of existing Open Education service also serve as a reference to ensure that each indicator 
is reasonably relevant. Table 5 illustrates reference from which each indicator synthesized from previous work. 
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Based on extensive literature review and analysis, we construct a set of criteria and indicator as a reference for 
assessing Open Education service quality (see Table 6). Each indicator also can be viewed as a factor to be 
considered when creating Open Education service. It is fair to say that these indicators might not give a complete 
coverage to service quality. However, we proposed that such indicators will serve as an initial step toward 
discussion of a more complete and detailed service quality measurement. Together with the evaluation scheme, 
this set of indicator form as an Open Education Metric which is expected to describe the quality of Open 
Education service in quantitative and quantitative manner. 
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Table 6. List of Criteria and Indicator 


Criteria 

Indicator 

Openness 

1.1. The service can be accessed by everyone 

1.2. All features freely accessible 

1.3. Service available in international language 

Benefit 

2.1. There are comprehensive education services 

2.2. There is certification or credit granted for learning effort or outcome 

Delivery 

3.1. Learning materials are classified based on the field of science 

3.2. Description about a course is available 

3.3. There is a definition about the aim and expected outcome of a course 

3.4. There is an information about knowledge or technical prequisites for a course 

3.5. List of scientific resource related to a course is available 

3.6. Information about learning evaluation mechanism for a course is available 

3.7. Detailed schedule of learning activity for a course is available 

3.8. Information to obtain guidance about a course is available 

3.9. There is a mechanism to motivate student in finishing a course 

3.10. Structure of each course designed in consistent manner 

Learning 

4.1. There is a wide variety of learning materials 

4.2. A course is structured into several topics as learning units 

4.3. Topics of a course directly related to the course title and description 

4.4. Topics sequentially arranged based on logical ordering 

4.5. Resource available for each topic/course directly related to its topic/course 

4.6. Facility for communication and discussion is available 

4.7. The use of communication and discussion facility is encouraged throughout learning process 

4.8. Learning is personalized to cater individual learner characteristics 

Evaluation 

5.1. There are materials to evaluate learning outcomes 

5.2. Evaluation material is available for each course 

5.3. Evaluation material properly address the content and objective of a course 

5.4. Evaluation is conducted to assess the learning outcome in each course 

5.5. The service provide feedback to learner based on evaluation of learning outcome 

5.6. Feedback given within a reasonable or definite amount of time since the evaluation conducted 

5.7. There is a standard of learning outcome in each course 

5.8. There is a mechanism for learner to give feedback to service provider 

Support 

6.1. Features for searching by keywords are available for: materials, subjects, and forums 

6.2. Service provide recommendation of related course or learning resource 

6.3. Technical support is available 

6.4. Learning tutor for a course is available 

6.5. Profile of course or resource author is available for learner’s reference 

6.6. Course is created and guided by a competent tutor or teacher 


Evaluation Scheme of the Quality of an Open Education Service 

In order to make the indicators of the metrics previously defined able to represent the quality of an Open 
Education service, it required an evaluation scheme. In this study, we have proposed a scheme on a qualitative 
assessment of each indicator by classifying the applicability of each indicator into four different levels that do 
not overlap. 


Broadly speaking, four of these levels are Level 0: Not Performed (not available), Level 1: Initial (already begun 
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to be applied to a small extent), Level 2: Delivered (mostly already applied), and Level 3: Managed (already 
fully implemented). Qualitative evaluation is done by comparing the existing implementation of a service against 
the circumstances described in each level of an indicator. Table 7 below contains a specific description of each 
level of the application on each indicator. 


Table 7. Examples of a Qualitative Assessment Scheme of Open Education Metric 


Criteria 

Indicator 

Level 0 

Level 1 

Level 2 

Level 3 

Openness 

1.1 The service can 
be accessed by 
everyone 

Can only be 
accessed if 
becoming a 
member of an 
education 
institution 

Yes, but not 
for all learning 
materials 

Yes, for all 

learning 

materials 

Yes, for all learning 
materials, with the 
right to modify and 
redistribute 

Benefit 

2.1 There are 
comprehensive 
education 
services 

Standalone 

and 

unstructured 

materials 

Unstructured 
materials 
suitable for a 
learning 
purpose 

Structured 
learning 
materials, 
packed per 
subject of 
interest 

Structured learning 
materials 

complemented with 
a learning activity 
as an online class 

Delivery’ 

3.1 Learning 
materials are 
classified based 
on the field of 
science 

No 

classification 

exists 

Classification 
exists for 
small fraction 
of all learning 
materials 

Classificatio 
n exists for 
most of all 
learning 
materials 

Classification exists 
for all learning 
materials 

Learning 

4.1 There is a wide 
variety of 
learning 
materials 

Only textual 
learning 
materials 
available 

Multimedia 
learning 
materials 
(video/audio/si 
mulation) 
available for a 
small fraction 
of subjects 

Multimedia 
learning 
materials 
(video/audio 
/simulation) 
available for 
most of 
subjects 

Multimedia 
learning materials 
(video/audio/simula 
tion) available for 
all of subjects 

Evaluation 

5.1 There are 
materials to 
evaluate learning 
outcomes 

No evaluation 

materials 

available 

Evaluation 
materials 
available for a 
small fraction 
of subjects 

Evaluation 
materials 
available for 
most of 
subjects 

Evaluation 
materials available 
for all subjects 

Support 

6.1 Features for 
searching by 
keywords are 
available for: 
materials, 
subjects, and 
forums 

All is not 

available 

Yes, only one 
out of the 
three items: 
materials, 
subjects, and 
forum 

Yes, only 
two out of 
the three 
items: 
materials, 
subjects, and 
forum 

Yes, all of the three 
items: materials, 
subjects, and forum 


Open Education metric also designed to be able to map a service into concepts in Open Education paradigm 
(Open Content, Open Courseware, etc.). The objective is to assign a proper category so that it represents 
characteristic of a service in general. Based on assessment of indicator 1.1, we can categorize whether a service 
is an open or closed service, which means whether it is open for anyone to participate or only allow a closed 
group of people to access. Then, based on assessment of indicator 2.1, we can categorize whether a service 
belong to “Content”, “Courseware” or “Teaching” provider. In context of indicator 2.1, Level 0 and 1 represent a 
content provider, Level 2 represent a courseware provider, while Level 3 represent a teaching provider. 

Furthermore, to establish a metric that also can provide quantitative information, we have proposed a 
quantitative assessment scheme. The quantitative scheme is basically made by assigning a weight for each level 
of implementation and a maximum score for each indicator. The value of service quality on a particular indicator 
is calculated by multiplying the weight with the indicator’s maximum score. The value of service quality on a 
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particular indicator is calculated by multiplying the weight with the indicator’s maximum score. The weight at 
every level of implementation serves as a multiplier so that the indicator’s value represents the real conditions of 
the Open Education service. 

Currently, we can only assume that each of the criteria has the same contribution to the overall quality of service, 
which is why each criterion has the same maximum value. Scores for individual indicator of each criterion are 
calculated by dividing the maximum score of each criterion by the total number of indicators related to a 
particular criterion. This scheme’s calculation model ensures that the maximum value that can be obtained for 
any service remain the same, although there is a change in the total number of indicators or levels. In this study, 
the proposed maximum value that can be obtained for a service is 1000. Table 8 shows the example of 
quantitative assessment scheme details on the Open Education Metric. 


Table 8. Example of the Quantitative Evaluation/Scoring Scheme 


Criteria 

Indicat 

or 

Numb 

er 

Weight 

Score of 
Maximum 
Indicator 

Score of 
Maximum 
Criteria 
XWeight*Score 

Level 0 
(min-max) 

Level 1 
(min-max) 

Level 2 (min- 
max) 

Level 3 (min- 
max) 

Openness 

n 

0.00-0.25 

0.26-0.50 

0.51-0.75 

0.76-1.00 

55.56 

166.68 

1.2 

0.00-0.25 

0.26-0.50 

0.51-0.75 

0.76-1.00 

55.56 

1.3 

0.00-0.25 

0.26-0.50 

0.51-0.75 

0.76-1.00 

55.56 

Benefit 

2.1 

0.00-0.25 

0.26-0.50 

0.51-0.75 

0.76-1.00 

83.34 

166.68 

2.2 

0.00-0.25 

0.26-0.50 

0.51-0.75 

0.76-1.00 

83.34 

Delivery 

Distribute score evenly across indicators of the same criterion 

1000/6 

Learning 

Distribute score evenly across indicators of the same criterion 

1000/6 

Evaluation 

Distribute score evenly across indicators of the same criterion 

1000/6 

Support 

Distribute score evenly across indicators of the same criterion 

1000/6 

Maximum Service Score : 

-1000 


The appropriate quantitative assessment scheme, is that the higher the level of implementation of the service, the 
higher the scores. Each of the indicators on the same criteria have the same maximum score. Each criterion has 
the same contribution to the total maximum value. The difference in the maximum value of the criteria at the 
second decimal digit occurs because the indicator score is rounded to two decimal points to simplify the 
presentation. The difference of the maximum value calculation scheme happens because of the same thing. 
Nevertheless, the differences are relatively insignificant on value criteria and the overall value is not more than 
0 . 01 %. 


Testing and Evaluation 

In line with the purposes of an Open Education Metric, which is to measure the quality of OE services as well as 
classifying them into a relevant Open Education category, we tested the metric applicability to assess existing 
OE services in the real world. The outcome of this evaluation includes the score and category of the OE service 
evaluated. The score can be viewed in overall or in detail per criterion/indicator, which depends on viewpoint 
intended. The category of an OE service represents which category it belongs to in Open Education concepts, 
which should be one of Open Teaching, Open Education Resources, Open Courseware, or Open Content. 
Assessment of the OE service quality was conducted manually, facilitated by the evaluation system. The OE 
services tested in this study hosted by two reputable educational institutions in Indonesia and have considerable 
number of materials available. Three of the OE services tested are: 

1. Institution A’s Open Courseware 

2. Institution B’s Massive Open Online Courses (MOOC) 

3. Institution B’s Enrichment Material 

Then, based on the evaluation result of each OE service tested, we can rank them according to the quality score 
obtained. Table 9 shows the ranking, the total score, and the category of each OE service tested. This score 
represents the quality of each OE service toward ideal Open Education implementation. 
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Table 9. Example of Ranking of OE Services based on Evaluation Results 


Ranking 

Service 

Total Score 

Category 

1 

Institution B’s MOOC 

705.91 

Open Teaching 

2 

Institution A’s Open Courseware 

523.97 

Open Courseware 

3 

Institution B’s Enrichment Materials 

511.13 

Open Content 


Figure 5 shows the comparison of the quality of the three OE services tested in this study. The classification of 
OE services into categories is not intended to represent a hierarchy, but only to indicate the nature of OE services 
provided. Thus, it should not be interpreted that one category better than other categories in every aspect. For 
example. Institution B’s Enrichment Material perceived better in terms of openness than Institution A’s Open 
Courseware. This is because the first one provides open access for everyone to all of its materials, in comparison 
to Institution A’s Open Courseware that only allow public access to a small fraction of subjects. Another 
example. Institution B Enrichment Material perceived better in terms of evaluation rather than Institution B’s 
MOOC. This is because each material in Institution B’s Enrichment Material is always accompanied by an 
evaluation rather than online classes in Institution B’s MOOC which does not always provide an evaluation for 
each subject. 



Figure 5. Example of Comparing OE Services Quality within Evaluation System 


Performance of the Open Education Metric as a measurement tool can be reviewed from several aspects, such as 
applicability, clarity, and quantifiability. This review is considered based on metric performance in assessing 
quality of existing OE services that we did previously. Table 10 provides the description of the Open Education 
Metric review against these three aspects. 


Table 10. Qualitative Evaluation towards Open Education Metric Designed 


Aspect 

Explanation 

Applicability 

Evidently, the Open Education Metric is able to assess the quality of OE services in the real 
world, with the score and category of OE services as the ultimate viewpoint. 

Clarity 

It can be observed that the Open Education Metric provides a set of indicators with clear 
intention, and a non-overlapping indicator level which each assessor to determines the 
appropriate implementation level for an OE service evaluated 

Quantifiability 

The Open Education Metric is able to translates OE services quality into a quantitative score, 
which is useful for the benchmarking and ranking of existing OE services. 


Expert judgment data were obtained by using a questionnaire in a form of Likert-type items which contain 37 
questions related to the relevance of each indicator to determine the quality of OE services. Each question 
consists of three types of responses: agree, neutral, or disagree. If an expert votes agree then it can be interpreted 
that the indicator is relevant to indicate OE service quality, and vice versa. In this study, we have obtained 
judgment from seven experts, and can be concluded that they all agree with the relevance of most of the 
indicators. 

User judgment data were obtained with the same set of a questionnaire used in expert judgment collection. In 
this study, we have obtained judgment from six users and can be summarized that most of the indicators 
considered to be relevant by most of the users. 


Copyright © The Turkish Online Journal of Educational Technology 


33 









































































































TOJET: The Turkish Online Journal of Educational Technology - July 2017, volume 16 issue 3 


CONCLUSIONS 

One of the main issues in the development of Open Education (OE) services is the availability of the quality 
assurance mechanism. This research aims to proposed a guideline to evaluate the quality of OE services. This 
evaluation result is important as a reference in the development of services, for evaluating existing services, and 
for benchmarking to compare the quality among OE services. 

Open Education Metric proposed consists of criteria, indicators, and the evaluation scheme. There are 37 
indicators proposed and divided into six criteria: (1) Openness - 3 indicators; (2) Benefit - 2 indicators; (3) 
Delivery - 10 indicators; (4) Learning - 8 indicators; (5) Evaluation - 8 indicators; (6) Support - 6 indicators. 
Each criterion represents different aspects in viewing the quality of OE service, while each indicator of a 
criterion elaborates related aspect into specific point. Each indicator designed in such a way that it can be 
independently observed on OE service via internet. In this study, each criterion has been assumed to have equal 
contribution to the overall OE service quality, while each indicator has the same contribution to its criterion 
score. The maximum score proposed for each OE service is 1000, in order to make it consistent and comparable. 

Furthermore, we conducted evaluation based on the Open Education Metric to test its performance in evaluating 
OE service quality in the real world. Three of OE services evaluated including Institution A’s Open Courseware, 
Institution B’s MOOC, and the Institution B’s Enrichment Materials. The evaluation results including score and 
category that indicate the overall quality of each OE service. The advantages and disadvantages of each OE 
services can be inferred to in detail by doing an analysis per criterion or per indicator. These results show that the 
Open Education Metric is capable to evaluate OE service quality into quantitative notion, while providing the 
adequate description of quality in detail. 
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